Imagery and annotations

ABSTRACT

A decade from now, a visit to the supermarket will be a very different experience than the familiar experiences of decades past. Product packaging will come alive with interactivity—each object a portal into a rich tapestry of experiences, with contributions authored by the product brand, by the store selling the product, and by other shoppers. The present technology concerns arrangements for authoring and delivering such experiences. A great variety of other features and technologies are also detailed.

RELATED APPLICATION DATA

This application is a continuation of application Ser. No. 14/152,925,filed Jan. 10, 2014 (now U.S. Pat. No. 9,818,150), which claims priorityto provisional applications 61/809,254, filed Apr. 5, 2013, and61/818,839, filed May 2, 2013. These applications are incorporatedherein by reference.

BACKGROUND AND INTRODUCTION

Shoppers presently can use their smartphones to scan product barcodes instores, to obtain product information. This capability is expected soonto be available with headworn apparatus, such as Google Glass.Applicant's published patent application 20130223673, and IBM'spublished patent application 20140002643, detail how such productinformation may be presented using augmented reality (AR) techniques.

AR techniques commonly work by recognizing an input image, andidentifying (and presenting) an augmentation that corresponds to thatimagery. One application of AR is in promoting movies. A user is invitedto capture an image of a movie poster (e.g., for the movieTransformers). A corresponding application on the user's smartphonerecognizes the poster artwork, and overlays a rendering of a 3Dcharacter from the movie (e.g., a Transformer who breaks out of theposter and starts a rampage).

Use of AR in retail shopping is expected to be less thrilling but moreuseful. For example, such a capability will allow shoppers to obtainingredient lists and discount information. Social network integrationwill enable presentation of information gleaned from the shopper'ssocial network friends, including their personal product reviews.

Prior AR implementations have been based on 2D imagery, e.g., triggeringthe corresponding annotation, and placing it—with respect to features ofthe 2D artwork. But packaging is three dimensional. And productpackaging often warrants more than a single annotation.

In accordance with one aspect of the technology, multipledifferently-oriented views of a 3D retail product package (e.g., acereal box) are captured during an enrollment phase. (A camera used tocapture the images may be fixed, and the package may be positioned on arotating turntable.) This collected imagery is then processed toidentify landmarks on different faces of the product. The landmarks maybe robust image features (e.g., SIFT points), or an invisible Cartesiancoordinate system that is defined by an array of steganographicwatermark tiles encoded in the package artwork. In some embodiments, thecaptured views are processed to yield a 3D model of the product.

This information is then used as a basis for authoring plural ARaugmentations for the product. For example, a marketing designer at thecompany that produced the product (e.g., General Mills) may definedifferent augmentations respectively linked to (1) the General Millslogo on the front of the box, (2) the Wheaties logo on the front of thebox, (3) the athlete's picture (e.g., Michael Phelps) on the front ofthe box, (4) the Nutrition Facts panel on the side of the box, (5) atext blurb about Michael Phelps on the back of the box, etc. etc. Thedesigner simply selects (e.g., by graphical highlighting) the region ofthe box that is to be associated with a particular augmentation, andprovides a link to the desired action.

Software translates the designer-highlighted region into coordinatesthat can be machine-recognized (e.g., using SIFT feature points, orsteganographic watermark data). Corresponding information identifyingthe region, and the designer-determined annotation that is to beassociated with that region, are stored in a database.

When a consumer later captures imagery of the cereal box, the product isrecognized, and a variety of different annotations arepresented—depending on the viewpoint. Exploring the product—byphysically moving the package, or by interacting with an on-screen userinterface—reveals additional annotations.

The annotations needn't all be defined by the product producer. Someannotations may be defined by the store in which the product is sold(e.g., Safeway). Other annotations may be defined by consumers. ADarwinian process can proceed, in which the most favored annotations arethose that are routinely presented.

The present technology thus enables product packaging to come alive withinteractivity—each object a portal into a rich tapestry of diverseexperiences.

The foregoing and additional features and advantages of the presenttechnology will be more readily apparent from the following detaileddescription, which proceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1H show views of a cereal box from different viewpoints, withdifferent annotations.

FIG. 2 shows a rendering of nutritional information.

FIGS. 3A and 3B illustrate how a feature on a cereal box can link to alabeled graphic, which changes in viewpoint presentation with changes inviewpoint of the cereal box.

FIGS. 4A-4C show some of the layer information stored about a product(i.e., a box of Wheaties cereal) in a database.

FIGS. 5 and 6 show some of the other information about Wheaties cerealin the database.

FIGS. 7 and 8 show views of exemplary nutrition-comparison applications.

FIG. 9 shows an illustrative symbol to denote a Living Package.

DETAILED DESCRIPTION

FIGS. 1A-H show views of a cereal box 12 from different orientations. InFIG. 1A, the front panel of the box is angled approximately 15 degreesrelative to the camera viewpoint. That is, the box has been turned about15 degrees in a clockwise direction from a frontal view, if the camerawere originally fixed in front of the box. (Or, conversely, if the boxis fixed, the camera has moved about 15 degrees in a counterclockwisedirection from an initial camera position in front of the box.)

The box 12 is shown as rendered on a smartphone display screen, e.g.,presented by a smartphone “app.”

In FIG. 1B, the front panel is angled approximately 20 degrees relativeto the camera viewpoint. At this orientation, the displayed cereal boxis presented with a rendered annotation 14 a—a rectangle that has ascript-G (the General Foods trademark logo) and the word “Nutrition.”This annotation is associated with a feature on the box, namely theNutrition Facts block found on the box's side panel 15. This annotationcan be added to the displayed information by the software app, orotherwise; e.g., it may be provided with imagery provided to the userdevice from a remote source.

If the user selects this annotation (e.g., by tapping it on thetouchscreen), a rendering of the Nutrition Facts graphic from the cerealbox is present in the smartphone UI, as shown in FIG. 2. Varioustransition effects can be employed to dynamically introduce thisdisplay. For example, it can expand out from one of the corners of theFIG. 1B display, or be wiped vertically or horizontally across thatdisplay. The user can return to the FIG. 1B display from the FIG. 2display by a suitable signal, such as tapping on the screen, or byanother gesture.

Note that while the feature with which the annotation 14 a is associated(i.e., the Nutrition Facts block printed on the side panel) was visiblein FIG. 1A, annotation 14 a was not included in FIG. 1A. It wasn't untilthe box viewpoint was further rotated, to the FIG. 1B perspective, thatthe annotation 14 a was presented.

In FIG. 1C, the cereal box has been turned further (or the camera hasbeen moved further), so that the front of the box now faces at an angleof approximately 50 degrees relative to the camera viewpoint. Note thatthe annotation 14 a has followed the box, i.e., it has moved slightly-infrom the border of the display screen, as the side of the box movestowards the center of the screen.

FIG. 1D shows the box further rotated relative to the camera, so thatthe front of the box now faces at an angle of approximately 60 degreesrelative to the camera viewpoint. In this figure a second annotation, 16a, has appeared. This second annotation is associated with a secondfeature on the box, namely a text block detailing the General Foodsguarantee of quality. This second annotation is labeled “Coupon” and, ifselected, presents the user with a coupon that can be redeemed for apremium, such as a discount on a purchase of a box of General Foodscereal.

FIG. 1E shows the box still further rotated, now to about 85 degreesaway from a frontal view. Again, the annotations 14 a, 16 a follow thebox, moving in slightly from the edge of the screen display.

FIG. 1F continues this rotation further, to about 95 degrees from afrontal view. Notice that while the second (“Coupon”) annotation 16 ahas remained on the right side of the box, the first (“Nutrition”)annotation has moved to the left side of the box (and been relabeled 14b). FIG. 1G shows a still further rotation (e.g., to about 110 degrees),in which the second (“Coupon”) annotation has joined the firstannotation in flipping to the left side of the box. (The secondannotation is here labeled 16 b.)

Flipping one annotation to the other side first, and later the secondannotation—instead of flipping both to the other side simultaneously—ismore appealing aesthetically. Functionally, it also draws attention tothe two annotations—particularly the first one to move.

FIG. 1H shows the box after further rotation. At this orientation (e.g.,about 125 degrees from the frontal presentation), a third annotation 18a (labeled “Recipe”) comes into sight, associated with a feature on theback panel of the cereal box.

Although not shown, as the box continues to rotate, annotation 16 bdisappears first, and then with still more rotation, annotation 14 balso disappears. Meanwhile, more annotations appear, as the rear panelof the cereal box comes more squarely into view.

As may be evident from the introduced appearance of annotation 14 a inFIG. 1B, and the later introduced appearance of annotation 16 a in FIG.1D, different annotations can be arranged to appear at different viewingangles—even if associated with features that are found on the same panelof the box (i.e., co-planar features).

In one implementation, there is a threshold angular value datum that isstored in a memory, associated with each annotation, defining theviewing orientations at which that annotation should be rendered. Forexample, this datum may specify that the Nutrition annotation 14 ashould be presented when the side panel 15 faces at an orientationwithin +/−78 degrees of the viewpoint. (That is, when the angle betweena surface normal of the side panel, and the camera viewing axis, is 78degrees or less.)

For the Coupon annotation 16 a, this threshold is smaller. Its storedthreshold angle data may specify that this annotation should appear onlywhen this viewing angle is 35 degrees or less. (Or, stated another way,this annotation should not be displayed when the angle between thesurface normal from the side panel of the box and the viewing directionexceeds a stored threshold value of 35 degrees.)

By such arrangement, different annotations can be given differentprominences. Some annotations may have very narrow viewing angles,requiring careful manipulation of the product (or the camera viewpoint)to find and reveal them. Other annotations are more generally viewable,from a diversity of viewpoints.

It will be noted that the “Nutrition” annotation did not changeappearance when it flipped from one side of the box to the other (i.e.,14 a and 14 b). However, in other embodiments, it may be desirable toalter the presentation of an annotation as its location moves. Forexample, the placement of the “G” (General Foods) logo on the left sideof the annotation may be flipped to the right side of the annotation,when the annotation is moved to the other side of the box. By sucharrangement, the “G” logo is always adjacent the depicted box. In bothcases, the same information content is conveyed (i.e., the logo and theword “Nutrition”), but the arrangement is different. Such twoannotations are an example of what may be termed twin annotations.

As indicated, differently oriented views can be presented on the screenof a device that is moved to obtain the different views from a physicalobject (e.g., akin to a viewfinder, or augmented reality display). Orthey can be presented on the screen of a stationary device, in front ofwhich an item is moved to present different views to the camera. Or sucha stationary device can render information provided to the device fromelsewhere, such as from another device or system (e.g., the cloud). Inthis latter arrangement, the user may navigate among the various viewssuch as by swiping on a touchscreen presenting the views, or byoperating a video navigation UI (e.g., such as a shuttle control).

Annotations are typically tied to anchoring features. These can be ofvarious types. For example, the nutrition panel on a food product may beregarded as an anchoring feature. Or a particular location in such anutrition panel (e.g., the top left corner of the “F” in the “NutritionFacts” heading) can be regarded as an anchoring feature. At the otherextreme, the whole front (or side) panel of a box may be regarded as ananchoring feature.

An anchoring feature can also be defined by reference to a spatialcoordinate in an item's geometry, such as a point two inches down fromthe top of the depicted Wheaties box, and one inch across the side panelthat is to the left of the front panel (when viewed from the frontpanel).

An anchoring feature can also be defined by reference to asteganographic reference system. For example, if the Wheaties box issteganographically encoded with a tiled pattern, where each tilecomprises 128×128 watermark elements (“waxels”), and the tiles are eachencoding with a payload that includes a tile number (e.g., in anacross-then-down numbering order), then an anchoring feature may bedefined as waxel {63,1} in tile 23. (The artisan is presumed to befamiliar with waxels, e.g., as further explained by patent publication20110214044, and by references cited therein. Tiling of watermarkpatterns across a cereal box is discussed and shown in pendingapplication Ser. No. 13/946,968, filed Jul. 19, 2013, now published as20140052555.)

(An anchoring feature may also be termed a “hotspot” on the package. Itmay be a point on the package, or a region of any size.) It will berecognized that the decision to present an annotation need not betriggered by detection of the corresponding anchor point in the imagery(e.g., by feature recognition). This decision can instead be triggeredbased on information about the viewed pose of the package, which may bediscerned based on a part of the package unrelated to the anchor point.For example, the decision to present the Nutrition Facts annotation ofFIG. 2 may be made based on pose information discerned from imagery ofthe front panel of the cereal box.

A software app that presents the user interface of FIG. 1 may belaunched in response to detection of a box of Wheaties in the field ofview of a user's camera-equipped device. For example, a user may be in asupermarket, and a Wheaties box on a shelf may be recognized by analysisof imagery captured by a smartphone carried by the user, or by a headmounted camera system worn by the user. (Thus, just as the annotationsshown in FIG. 1 can be the origins of links to other information andactions, the FIG. 1 arrangement—itself—can be the destination of such alink, originated from elsewhere (e.g., from applicant's Discover app, asdetailed in published patent application 20120284012).)

Once the user device has recognized the physical cereal box in thesupermarket (e.g., by techniques such as digital watermarking or imagefingerprinting—such as SIFT), the user can explore the annotationsassociated with the box. There are various ways in which this can occur.One involves applying the annotations over live imagery captured by theuser device camera. That is, the user can manipulate the physicalproduct in front of the camera. As different views are recognized (e.g.,again by analysis of fingerprint or watermark information), differentannotations can be caused to appear overlaid on the live camera imagery.(The artisan will recognize that steganographic digital watermark dataencoded in the artwork of printed packaging—such as on a cerealbox—allows the pose of the box relative to the camera to be determined.Briefly, this is because the geometrical scale and orientation of thewatermark information is known in advance (e.g., all items may beencoded with tiled watermark patterns that each comprises an array of128×128 waxels, at a scale of 75 waxels per inch, with the tilingstarting in the upper left corner of the item's panel). When sensed by acamera, the distortion of the sensed watermark pattern from its knownconfiguration permits the viewing distance and other pose parameters tobe determined. If reference SIFT data is known for the cereal boxartwork, the box pose relative to the camera can similarly be deduced.)

A second way of exploring the annotations involves ignoring the liveimagery captured by the user device camera, and instead presenting asequence of still images (or video) provided from a remote computersystem, e.g., across the internet, through which the user can navigate.

In this latter case, the views of the item may be authored by a thirdparty. This third party can be a commercial service provider thatspecializes in capturing and distributing product imagery, such asGladson, LLC (Lisle, Ill.) or ItemMaster, LLC (Skokie, Ill.). Thesecompanies typically are provided physical samples of each new productintroduced by retail manufacturers, and capture a series of promotionalimagery—such as by placing the item in a robotic studio thatsequentially captures images at different product poses. For example,the Wheaties box of FIG. 1 may be photographed at every 10 degrees ofhorizontal rotation.

If the box is watermarked, the watermark information in the capturedimagery can be decoded, so as to identify the placement of differentwatermark tiles at different locations on the box. (The payload of eachwatermark tile may include the GTIN identifier of the retail product,and the serial number of that tile on the box.) Map data can thus beproduced, e.g., permitting each physical location on the box to becorrelated with a corresponding coordinate in the watermark tile space(e.g., tile 23, waxel {63,1}). This map data can then be provided backto the retail manufacturer (e.g., General Foods), which can author theannotations that should be associated with different features on thebox. General Foods may specify, for example, that a Nutrition annotationshould be associated with watermark tile 8, waxel {23,55}, and that itshould be rendered when within an angular viewing cone of 70 degrees.Similarly, it may specify that a Coupon annotation should be associatedwith watermark tile 37, waxel {110,63}, and that it should be renderedwhen within an angular viewing cone of 32 degrees. This information,together with associated link data (e.g., a URL to a Nutrition graphic,or to a video), is stored in a database where it can be accessed by theuser device.

In some cases, annotation authoring can be as simply as identifying acontent item (e.g., a graphic or a video) within a digital assetmanagement (DAM) system used by General Foods to organize its mediacontent.

The image data captured from the product sample can also be analyzed toderive SIFT feature information, by which the item can be identified.Map data can also be produced that permits each physical location on thebox to be identified in terms of its placement relative to the SIFTfeatures. Again, annotations can be authored on this basis.

The robotic processing of product samples by companies such as Gladsonalso typically gathers accurate product dimensions and other productshape/configuration data. From such information a 3D model of theproduct can be derived, in the form of a mesh, wire frame, or otherwise.Since the imagery is captured in the same process, the imagery can becorrectly registered with the 3D model data. Gladson has announced apartnership with Red Dot Square to produce such 3D model data. (Asdetailed in application 61/838,165, filed Jun. 21, 2013, a variety ofcommercial software packages, including Autodesk 123D-Catch, and CubifyCapture by 3D Systems, Inc. (formerly Hypr3D), can alternatively be usedto create 3D model data from plural images.)

This product model information can be stored in the noted database andprovided to the user device, along with the other noted information,upon detection of a Wheaties box in a supermarket. This allows the userto explore a virtual rendering of the product on the user device—not byexamining selected images (e.g., every ten degrees horizontally), or bynavigating a fixed video sequence, but instead by using a 3D UI toexplore the item in a manner of the user's choosing. Again, annotationsappear in the manner defined by the manufacturer.

Thus, there are several possible viewing modes. In one mode, after theobject has been recognized, the screen can show the object as sensed bythe device camera. In a second mode, after object identification, thescreen can disregard the physical scene, and present instead a sequenceof stored images (or a video) of the object depicting different views,through which the user can navigate backwards and forwards, e.g., byswipes or other gestures. In a third mode, the physical scene is againdisregarded, and the screen presents a rendering of a 3D model of theobject, around which the user can navigate by gestures or otherwise. Ineach such mode, annotations can be added to the displayedinformation—either by the local device, or a remote device (e.g., thedevice that provides the stored images or 3D model). Sometimescombinations of such modes may be employed (e.g., the object may besegmented out of the imagery captured by the device camera, and arendering of a 3D model can be presented—surrounded by the visual scenein which the object was found).

As just noted, in the case of a user device interacting with a physicalitem, it is typically necessary for the device to identify the item(unless the item is identified otherwise, such as by an indication fromthe user). Once the item is identified, the associated annotations canbe downloaded to the device, such as from a remote database record thathas been located using an item identifier. Each annotation is associatedwith parameters that define the circumstances when it should bedisplayed, such as the angular threshold referenced above.

The circumstances can also involve context—including user context andcomputational context. For example, if a particular user has a historyof never selecting annotations that reveal recipes, then over time theuser device can learn not to render recipe-related annotations.

Of course, this technology is not limited to cereal boxes, and is notlimited to experiences that are commercially authored. Anotherembodiment concerns a user who is selling a car, such as on Craigslistor EBay. The user may take several photographs of the car. Using asoftware application, the user identifies a point in one of the photoswhere an annotation is desired, e.g., a dent that the user wants to callto the attention of prospective buyers. The application then invites theuser to type the information that should be associated with thatlocation on the depicted car (e.g., a note “This ding has an interestinghistory . . . ”), or to paste website URL to which buyers should be ableto link from that physical location on the car). The annotation canappear not just with the photo in which it was identified, but alsoother photos in which the dent (a feature anchor point) appears.

Data identifying the noted points in the pictures are stored in adatabase, together with the associated annotation data.

Other Remarks

While illustrated in the context of a physical, retail product (e.g., abox of Wheaties), the same principles can be applied to syntheticshapes, e.g., rendered from 3D image files. Similarly, the informationrevealed when a user taps the annotations shown in FIG. 1 needn't bestatic (such as the nutrition information shown in FIG. 2). Instead, theannotations can link to dynamic behavior, such as initiating othersoftware activity (e.g., launching other software or another app), ordisplaying a video. In a Wheaties box featuring a Tour de France rider,for example, an annotation can link to a live video showing the rider atthat very moment, as captured by a camera in the rider's support car. Orthe annotation can link to a map on which the cyclist's location at thatmoment is shown by a flashing red arrow. Etc.

In some implementations, the annotations needn't be links to onlydestinations from which corresponding “payoffs” are fetched andrendered. Instead, the annotations themselves can present the payoffinformation. FIGS. 3A and 3B, for example, show how a feature of thedepicted Wheaties box links to a labeled view 22 of a bicycle. The bikedepiction changes in size and perspective with different views of thebox, simulating that it occupies the same 3D space as the box, and issubject to similar geometrical perspective phenomena (e.g.,foreshortening, etc.) as the viewpoint of the box changes.

While reference was made to selecting the annotations by tapping ontheir respective screen graphics, other arrangements can alternativelybe employed. For example, if there is just a single annotation, tappingon the General Foods logo 24 in the lower left of the UI can serve toselect that annotation. If several annotations are present on thescreen, each tap of the logo can highlight a different one of theannotation graphics, e.g., progressing in a clockwise direction from theannotation graphic closest to the lower left corner. When the desiredannotation is highlighted, the user can touch and hold the logo 24 toeffect selection of that annotation.

Selection of annotations can be performed by means other than screentapping. Voice recognition, for example, can be used. If only oneannotation is visible, the user can select that annotation by speaking aword such as “select” or “go.” If several annotations are visible, theuser can select one by speaking a distinctive keyword, such as“nutrition” or “recipe.” Some annotations may be selectable by pluraldifferent keywords. For example, where an annotation presents severalwords or icons/illustrations, it may be selected by speaking any ofthose words, or any word descriptive of the icon/illustration.

It will be recognized that item-identifying information may bedetermined from sources other than image data. For example, a productidentifier may be read from an NFC (RFID) chip on a product, on a shelfdisplay, or elsewhere, using the NFC reader provided in manysmartphones. From image data, a barcode can also be decoded.

As noted, watermarks enable the pose of an object to be determined fromanalysis of image data. This pose information includes distance from thecamera to the object. (The apparent scale of the watermark diminisheswith increasing distance.) Thus, a conventional camera, imaging awatermarked object, and analyzing the watermark information, iseffectively a depth-sensing imager. The physical inclination ofwatermarked surfaces is similarly determined from the geometricaldistortion of the known watermark signal. This capability allows thephysical 3D configuration of a watermarked object to be discerned from2D image data. This is akin to the familiar “structure from motion”process of determining a 3D shape, but without any requirement ofmotion. A static image suffices. All of the technologies andapplications that use depth-based imagery, or structure derived fromimagery, can be practiced more simply by watermarking the subject, anddiscerning the depth/structure from the watermark information.

Actively Managed Consumer Packaged Goods

The following discussion elaborates on some of the above-discussedpoints, and details additional novel technology, e.g., concerning activemanagement of consumer packaged goods (CPGs).

A decade from now, a visit to the supermarket will be a very differentexperience than the familiar experiences of decades past. Packaging willcome alive with interactivity—each object a portal into a rich tapestryof experiences, with contributions authored by the brand, by the store,and by other shoppers. It all begins with the packaging.

A GIS (Geographic Information System) serves as a useful analogy. A GISis a system designed to capture, store, manipulate, analyze, manage, andpresent all types of geographic data. In some aspects, GIS technologymerges cartography, statistical analysis, and computer science toprovide an infrastructure for spatial data. So, too, do aspects of thepresent technology.

A GIS system typically provides many layers of spatial informationrelating to a geography. The regional government for Portland, Oreg.,for example, publishes an on-line MetroMap application, by whichgeographical information about the region is available for manipulationand exploration. A base map is selected and can comprise satelliteimagery, terrain information, and/or street map information. On top ofthe base information a user specifies desired layers that should bevisually overlaid. These can include, e.g., zip codes, school districts,tax lots, wetlands, vacant land, fire districts, etc., etc.

ESRI is a leader in the GIS market, and its ArcGIS, ArcView, ArcMobile,and ArcServer software tools are exemplary of the tools available inthis space.

The artisan is presumed to be familiar with GIS technology, includingtechniques by which disparate spatial data sets are stored, manipulated,and geometrically registered with each other.

Back to packaging, just as a GIS system integrates disparate informationrelating to a geographical region, so too does an implementation of thepresent technology integrate disparate information relating to a productand its packaging. FIG. 6 shows part of a simple database record thatintegrates some of this disparate information for the Wheaties box.

When General Mills introduces a new Wheaties package, it provides asample to Gladson. As noted, Gladson employs robotic apparatus tophotograph the package from multiple viewpoints, and to capture weightand dimension data.

From the captured imagery, the locations of features on the packagingare identified. These include the headline name/logo, the nutritionpanel, the ingredients list, the photographic image, the color splashesand ornamentation, the narrative text blurbs, etc., etc. Eachdistinguishable item, or class of items, may be stored in acorresponding layer of information (in a GIS sense) in the FIG. 6database record for the object.

For the illustrative Wheaties package, FIGS. 4A-4C show, respectively,layers for the headline name/logo, the photographic image, and thenutrition information. (Although not legible in FIG. 4C, the illustratedfront panel feature details that one serving of Wheaties includes 100calories, 0 grams of saturated fat, 190 mg of sodium, 4 grams of sugar,3 grams of fiber, and 45% of the recommended daily value of iron. ThisFIG. 4C layer may also include the Nutrition Facts graphic, which isprinted on the hidden right side of the package.) A package may have adozen or more such layers dedicated to artwork and other printedinformation on the package (e.g., net weight statement, expiration datestamp, box-top coupon, Facebook link, recipe, customer service contacts,corporate address, recycling information, etc., etc.). Each can beseparately detailed in the database record. (FIG. 6 omits much of thisinformation, for clarity of presentation.)

Stepping back to original creation of the package artwork, a programmedimage processor may have been employed to slather a pattern of tiledwatermarks across package artwork as an automated pre-pressoperation—after the artwork was otherwise composed. This pattern canserve as a coordinate system by which the locations of the overtfeatures can be determined.

The position of the watermark tiles, relative to features in theartwork, may be unknown to designers of the package art. But after thepackage has been assembled and filled with product, the imagery capturedby a party such as Gladson can be analyzed to determine the positions ofthe watermark tiles in the captured imagery, and this information can bestored in the database record. From this information, Gladson (oranother) can generate package map information that defines the spatialcorrespondence between artwork features on the package, and thewatermark tiles. (This spatial correspondence information can also bestored in the database record.) With such information, the marketingdepartment of General Mills can then associate differentannotations/promotions with different features of the package, tothereby author an interactive experience for the package. A humanoperator can define this interactivity by reference to visible features.Through the stored map information, these references can be translatedto refer to imperceptible watermark features (which are often moreeasily identified by software apps than visible features), or to areasdefined in terms of absolute {x,y} coordinates on the box. (In FIG. 6,the position information is given relative to the upper left hand cornerof the printed cardboard “blank” from which the box is assembled, but ofcourse other coordinate reference systems can be used.)

FIG. 5 shows this watermark map information stored in the database. Inthis example, an automated watermark embedding process placed the originof the tiled watermark pattern at the top left corner (not shown) of theflat package blank from which the 3D package is assembled (e.g., byfolding and gluing). The width of the blank is about 23 inches, whichrequires 14 side-to-side watermark tiles to span. The top part of theblank, folded to produce the top panel of the package, spans the firstrow of 14 tiles. The first tile that appears on the front face of thepackage is the bottom right part of tile 20, which is followed by thebottom parts of tiles 21, 22, 23, 24 and 25 across the top front edge.(Part of tile 25 folds over onto a side panel.) Parts of other rows ofwatermark tiles span successive horizontal stripes across the front ofthe package, as shown in FIG. 5. (Certain tile numbers are omitted fromthe figure for clarity of presentation.)

This package map information, defining the spatial correspondencebetween artwork features on the package and the watermark tiles, isstored in layer form in the database, e.g., using GIS-like dataconstructs. Such layer(s) can identify the location of each watermarktile, the data payload for each tile, the artwork encompassed, theannotation(s) associated with different location in the tiled pattern,etc.

In a simple example, an interaction designer in the General Millsmarketing department may specify that if a user taps anywhere on theartwork depicting Michael Phelps on the front of the box, such useraction should trigger launch of a 15 second Michael Phelps video fromwhich the still image on the cereal box was clipped. The software toolcan translate this reference, using the watermark map information, to atap anywhere in watermark tiles 91-94 or 105-108. (Fractional parts of awatermark tile can also be referenced, using waxel offsets from theupper left corner of a watermark tile.)

Fingerprint data for the object is also discerned. It, too, is stored asa further layer of information in the reference database record.Similarly, “Thingerprint” data is generated from the collection of imageviews, or from the 3D model, and this information is also stored.(Thingerprint technology is detailed in patent application 61/838,165and permits identification of 3D objects by profile features,Morse/morphological features, and image features.)

Other data layers store luminance and color information captured fromthe box in the Gladson image capture process. Likewise, 3D modelinformation that is derived from the Gladson-captured information isalso stored in the reference database record.

(While the foregoing described certain layers as being populated by datacaptured by Gladson, these layers—or others—can be populated from theorigination materials authored by the General Mills graphic designers,e.g., InDesign files, produced by Adobe's InDesign publishing software.Similarly, while the discussed layers generally contain graphicalinformation about a product package, other information—such as physicalinformation—can be provided as well. For example, one or more layers candetail the vendors of the different inks used, and their respectiveidentification numbers. Another layer can define the locations of gluetraces that were laid down to assemble the box. Other attributes may notbe spatial information, per se, but can relate to the object as a whole,i.e., global metadata. Examples include where the product was produced,the states into which it was shipped, the retailers known to stock theproduct, etc.)

The stored layer data serves as a baseline reference to which differentpackage functionalities can be mapped. The nutrition information ishere, and the General Mills logo is there, etc. These are locations towhich functionally-related augmentations (payoffs) are anchored, andrelative to which augmentation imagery can be draped on the model. (Anexample of draped augmentation imagery is where the photograph ofMichael Phelps on the front of the Wheaties box is masked, and a videois instead rendered in its place—the video being one from which thestill-image swimming photograph of Phelps was taken. This is indicatedby the “DRAPE” specification in the FIG. 6 database record, at 71.)

The augmentations may be referred to as “bubbles”—analogous to thedialog bubbles spoken by printed comic characters. The term is also usedherein synonymously with “annotations.”

As noted, a region on the package that elicits a certain annotation maybe referred to as a “hotspot” for that annotation. Hotspots can bespatially defined in various ways, including by reference to watermarkinformation for that region, by reference to fingerprint landmarks(e.g., scale- and rotation-invariant feature points, such as SIFT), byCartesian coordinates (e.g., of opposing corners of a rectangularregion), by graphical mask data stored in the database (defined usingvector graphics, or by artwork such as in FIGS. 4A-4C, or otherwise),etc.

While the foregoing has detailed certain aspects of preparing thepackage for interactivity (i.e., registering it in the system), thispreparation phase also extends to preparing the “pipes and otherplumbing” by which its interactivity will be delivered. This supportinginfrastructure needs to allow the brand, and/or other parties, to revisethe product interactivity once the product is out in the stores.

That is, the experiences associated with the Wheaties package desirablyshould not be defined once and fixed forever. Rather, they should beadaptable based on events and circumstances, including different adcampaigns, introduction of the product into different markets, userfeedback, different user and other contexts, retailer customization,etc. The augmentations should be tunable.

For example, an augmentation may be functionally associated with thephotograph shown in FIG. 4B, and defined—initially—in a generic manner,e.g., as Bubble-1. When the product is first released, data in thedatabase can specify that Bubble-1 should present the swimming videoclip from which the still image was excerpted. But if Phelps later doesa pool-side victory dance at an exhibition swim meet in Santa Clara,Calif., which received lots of local press coverage, then General Millscan revise the stored data to specify that, for the next thirty days,users in the Bay Area of California who interact with the photographshould instead be shown a video clip of Phelps' victory dance at theSanta Clara pool. Similarly, if a YouTube video featuring Phelps goesviral (e.g., the “Call Me Maybe” video featuring the US Olympic Teamswimmers), then the augmentation associated with the Phelps picture maybe temporarily defined to present the YouTube video.

Each generic bubble is thus associated with stored information detailingthe information and graphics that should be presented to a user thatselects a corresponding feature (hotspot) of the package forinteraction. This information can be stored in a globally-dispersedfashion. Just as General Mills caches locally-customized versions of itsweb site at servers around the world (e.g., a content delivery networkcontracted from Akamai Technologies), so too can augmentationfulfillment data for General Mills' products be stored inregionally-tailored, globally-dispersed fashion (e.g., replicateddatabase records). As marketing campaigns and other factors dictate, theaugmentation fulfillment data for each product bubble can be revised.

Moreover, the augmentation information delivered to consumers needn't bethe same for all consumers. As just-noted, consumers in some regions mayreceive augmentations different than consumers elsewhere. Similarly, theaugmentations delivered to a particular consumer can depend on personaldemographics, preferences, history, other context, etc. John and JaneDoe—who shop together for groceries while pushing toddler Tommy Doe inthe shopping cart—may leave the supermarket with different productinteraction experiences.

The augmentations needn't all be brand-defined. Just as conventionaltelevision broadcasts include a mix of promotional messages—some fromnational advertisers, some from the national broadcast network, somefrom regional advertisers, and some from the regional broadcaster, sotoo can CPG-related augmentations be defined by a variety of differentparties.

The brand (e.g., General Mills) may insist on control of allaugmentations associated with its trademarks and other proprietaryfeatures. But other aspects of the packaging may invoke augmentationsdefined by the retailer (e.g., Safeway). After all, the consumer istypically interacting with the product in physical space owned by theretailer, and may be using network services (e.g., WiFi) provided by theretailer.

In some embodiments, the augmentations viewed by the consumer depend onwhat software application is used to interact with the product. If theconsumer is using the Safeway shopping app, then Safeway's augmentationswill be those that are used by default. If a General Mills app is beingused, then it will naturally want to provide an experience that favorsits own content. Rules in the app (or in the server with which itcommunicates) can be processed to arbitrate between alternativeaugmentations, and decide which should be rendered to a given user at agiven time and place. These rules can work on data that includes allmanner of context information—not just the provider of the app (e.g.,Safeway or General Mills), but also the user demographics, location,past history, other context, etc.

Many retailers, and brands, employ digital asset management (DAM)systems for ingesting, cataloguing, storing, retrieving, and retrieving,product-related images, video and other media assets. Extensis, Razunaand North Plains are a few of the many vendors offering DAM technology.Such systems can be employed to serve as databases in which objectinformation is stored, and from which responsive augmentations areserved, etc.

From the foregoing it will be seen that, before a product is firstshipped to stores, various data layers associated with the packaging canbe defined, and generic augmentations can be functionally mapped todifferent features or locations (hotspots) in these layers. Data storageto receive fulfillment data for each of these augmentations is defined,and provisions are made to replicate such stored data throughout ageographical area in which the product may be distributed.

Once a product's baseline identity has been registered in the system(e.g., by watermark, fingerprint, etc.), and a template of itsinteractivity (e.g., functionally-mapped augmentations) has beenestablished, it is ready to be passed to the brand's creative team, forcreation of default interactions for some or all of the augmentations.

This takes us (back) to the authoring process—bringing the product tolife. Using a product identifier (e.g., watermark or fingerprint data),or otherwise, a designer opens up the data record(s) that defines thevirtual instantiation of the interactive product. An authoring tool mayrender this data to graphically show a 3D model of the product withblank bubbles—each functionally mapped to a product hotspot. Differentlayers may correspond to different bubbles (augmentations). The designernavigates around the 3D model, selects one of the bubbles and usesfamiliar web authoring tools (e.g., HTMLS, Java, etc.) to createcompelling content. Alternatively, the designer can map the bubble tofulfillment information already available in the brand's digital assetmanagement system, e.g., so that when a consumer selects theaugmentation, a stored video plays on a particular part of the devicedisplay (which may be defined relative to the position of the product,or product model, on the display). Augmentation defined in this fashionis then added to a corresponding database record.

To date, augmented reality has been used mostly as a parlor trick—eyecandy without lasting utilitarian value. The charter for creativedesigners is to avoid such pitfalls in use of the present technology,and provide augmentation responses that meet bona fide customer needswith enduring solutions.

Of course the sky is the limit on creativity. The far out musings of theedgiest designer will seem tired and dull five years out, as newfrontiers of technology are opened, and early-adopting creatives rushin. But again, enduring utilitarian value—not glitz—is the mantra.

One form of data that has enduring utility seems to be health-relateddata/statistics. The recommended daily allowance of salt andcarbohydrates is an example. These allowances can vary with age, genderand weight. While the figures published on consumer packaged foods are,of necessity, average values, the augmentations provided by the presenttechnology can be tuned so as to present data appropriate for the user'sparticular age, weight and gender.

Some health-conscious shoppers may be interested to view the nutritionfacts for Wheaties and other products in the context of the morestringent benchmarks established by the European Union, or thoseproposed by the US National Academy of Sciences but not yet adopted bythe US FDA.

For some augmentations, enduring utilitarian value comes fromcrowd-sourced, social information. Crowd-sourced product reviews onAmazon and TripAdvisor are generally regarded as useful; a shopper onEBay who disregards a seller's poor feedback ratings is asking fortrouble. In like fashion, crowd-sourced ratings and other commentary onconsumer packaged goods is expected to be a valuable tool. Morevaluable, still, is information from the shopper's own circle ofacquaintances—such as friends on Facebook and other social networks.

Thus, in accordance with another aspect of the technology, some productaugmentations may link to sites where reviews are posted. Such a site'suser interface can allow filtering to show, e.g., only feedback from theuser's friends; or from others in the user's zip code, county, or state;or from people who are otherwise demographically similar to theuser—such as by gender and age. The feedback from other users may,itself, be rated for utility (much as Amazon users' votes as to whichcustomer reviews are most useful), and the site's user interface maypermit filtering to show only feedback rated in the top ten percent ofall available feedback. And combinations of such filter criteria, etc.

In one particular implementation, a product may include an augmentationthat overlays the social network profile picture of a friend who hasrated a product, together with the text of that friend's productcommentary. With a quick gesture, the user device can place a phone callto that friend, or start an instant messaging session, so that user canpose clarifying questions to that friend.

A supermarket shopper who faces a bewildering array of pickles on theshelf may activate a feature in the software that causes each jar to berecognized, and overlaid with numeric ranking data (e.g., number ofstars, out of five) contributed by others. Where a friend has ranked aproduct, that friend's face may appear overlaid on the product. The facemay be bordered with a color indicating whether the review is positive(green) or negative (red). The numeric ranking data may be similarlycolor-coded. A quick scan of such information (which may be presented byheadworn apparatus such as Google Glass) may help the shopper quicklyovercome his initial bewilderment, and make an informed choice.

In some cases the crowd-sourced information isn't review data. Instead,it is simply a record of one or more product purchases, e.g., by aparticular friend, or by all friends in the aggregate. Such informationis known for all shoppers enrolled in a store's loyalty program and,with suitable permissions, can be shared with some or all othershoppers. For example, when Tom is picking up ice cream and chips tobring to a potluck hosted by the neighbors, he may configure thesoftware to graphically indicate which ice cream and chip varieties theneighbor evidently prefers, as indicated by purchase history.

(As evident from the foregoing, the interactivity supported by thepresent technology needn't be limited to isolated examination of oneproduct at a time, but can extend to methods that process views ofmultiple products at a time.)

Some consumers may find augmentations authored by other consumers to bemore useful than augmentations that the brands or retailers publish.Besides product reviews, this can include other consumer crowd-sourcedcontent.

Consider the Wheaties box, which may have a previously-definedaugmentation for the Nutrition Facts panel, but no augmentationspecifically linked to its statement of sodium content. A shopper who ison a personal campaign to increase awareness of salt in our diet maydefine her own annotation that is linked to the sodium entry on theNutrition Facts. She may snap a picture of the box side panel and, witha rudimentary authoring app, outline a rectangle around the sodiumentry. The app then prompts her to enter information, or a link, thatshould be used in presenting a corresponding augmentation. The shoppernavigates to an online article published by Prevention Magazine, “8Health Dangers of a High Sodium Diet,” and instructs the app to presentthe article title as a hyperlinked augmentation to future shoppers whoimage the Sodium entry on the Wheaties box. If such a future shoppertaps the displayed annotation, the article appears on their screen.

In like fashion, other shoppers can author augmentations that are mappedto this and other product features on the Wheaties box.

Over time, many alternate annotations for the Wheaties box sodium entrymay be authored by different shoppers. If there are two alternateaugmentations, both may be displayed in a two element menu within asingle augmentation bubble, and a shopper can elect to tap one, either,or none, to pursue further information. The same approach may be usedwith three or four alternative augmentations. But at some point thereare too many alternatives to practically display them all. In this case,a menu of annotations may present, e.g., three alternatives, and invitethe user to tap “More . . . ” to see the next, e.g., ten. The threealternatives that are presented in the augmentation bubble are thosethat are most commonly tapped by consumers for more information. Theaugmentation thus tunes itself so that the most popular (useful)crowd-sourced links rise to the top of the list, and are the ones mostprominently presented as available alternatives.

Of course, the same sorts of annotations can be authored for features onother products, throughout the retail universe. Eventually the number ofcrowd-sourced annotations may far exceed the number of brand- andretailer-authored annotations.

Data for these crowd-sourced annotations may not be stored in thedatabases employed by General Mills and other brands/retailers, but maybe stored in a Creative Commons or other public service data repository.Or it may be stored at web site dedicated to the particular product.Some implementations of the technology treat the different datarepositories (General Mills, Safeway, Creative Commons) as a federateddatabase, across which a single search can be done for relevantannotations. (Of course, the opposite—a single, unitary database inwhich all the information is physically consolidated—can also beemployed.)

Preference information stored by the shopper can pre-configure how thesoftware treats different or alternative augmentations, e.g., specifyingthat it should either show, or hide, crowd-sourced annotations.Similarly, the configuration may allow the shopper to otherwise specifythe types of annotations that are to be revealed or hidden, and otherconditions, e.g., limiting them in number, by subject matter (don'tbother me with coupons), source (don't show me stuff from nationaladvertisers unless they pay me at least a dime in checkout credit foreach impression), etc.

A Google Glass-wearing shopper scanning store shelves may configure thesystem to overlay faces on products recommended by those friends.Another may configure it to overlay faces on products that have merelybeen purchased by friends. Still another may configure the software tographically flag those products that have been reviewed negatively by afriend.

And such shopper preference information is not limited to treatment ofcrowd-sourced and social information. Shopper Geoff may indicate to thesoftware that he is not interested in coupons; Shopper Ed may show—byrepeated inaction—that he is not interested in recipes. Accordingly, theannotations presented to these users can be tuned to conform to theirexpressed or implied preferences.

Some crowd-sourced annotations are made by existing social networkinfrastructure—without any hosting by a brand or store. An example iswhen a consumer selects a Twitter augmentation presented for a product.This augmentation causes the user's device to load the Twitter app, andstart authoring a tweet with a hashtag indicating the product (e.g.,#Wheaties). The user can then author a comment about the product andpost it with a few further keystrokes. (This functionality canadditionally, or alternatively, present the tweets of others that relateto the product.) It will be recognized that some of the notedaugmentations are not thematically tied to any particular feature on thepackaging, but rather to the product itself (i.e., the stuff inside thepackaging). Such augmentations may be functionally mapped to theheadline name/logo, or they may be generally linked to the package as awhole, i.e., globally. For example, tapping or otherwise selecting thedisplayed product at a location not occupied by an augmentation/bubble,can trigger presentation of such globally-linked information.

It will further be recognized that some products don't have well-definedfeatures that can be spatially-mapped, and to which differentannotations can be anchored. Examples abound in the fruit and vegetableworld, e.g., fresh lettuce. Nonetheless, such products can be recognizedusing known pattern recognition techniques. (See, e.g., Rocha, et al,Automatic fruit and vegetable classification from images, Computers andElectronics in Agriculture 70.1, pp. 96-104, 2010; and Dubey, et al,Species and variety detection of fruits and vegetables from images,International Journal of Applied Pattern Recognition 1.1, pp. 108-126(2013), as well as published patent application 20130308045 and pendingapplication 61/907,362, filed Nov. 21, 2013.) Once recognized,corresponding augmentations can be identified and presented.

One class of bubble with which all objects can be equipped is a “liveassistance” bubble. When a user selects this option (e.g., by tapping onan augmentation), a communication channel is initiated with the customerservice department of the company that produced, or distributed, theobject. The user can then present—by text or voice or livevideo—whatever question or other matter needs attention (e.g., questionsabout food ingredients, warranty and return information, productinstructions, etc.). The customer service representative responds asappropriate, again by text or voice or live video.

Back to initial authoring of a product experience, it is expected thatthe creative process will include extensive focus group testing ofvarious augmentation alternatives. (Do you prefer this video of Phelpspulling away from his arch-rival during the race, or this other video ofhim after finishing, and seeing his record-breaking time on the raceclock?) Each demographically-tuned augmentation can also be A/B tested.(For the 30-40 year old set, are you more interested to hear that wholewheat might help avoid heart disease, or might help you lose weight? Forthe 50-60 year old set, is avoiding stroke more important than avoidingtype 2 diabetes? Etc.) If satisfaction metrics aren't high enough, newor additional design resources may be called-in to revamp theexperience. Eventually, a pre-live version will pass tentative muster,and be presented for sign-off by brand executives and perhaps CorporateManagement.

When the product experience finally goes live, campaign management toolsenable the brand to quickly make post-introduction changes as they arerequired, e.g., when new celebrity endorsements are made, or when a newuniversity study shows that a diet high in whole grains helps reduce therisk of Alzheimer's Disease. Back to re-write goes the design, withcreatives revising the augmentations to tout the latest findings—perhapswith more rounds of focus-group testing to assure maximum efficacy. Thedaily news cycle invades brand management, triggering daily revisions toselected product experiences in response to the tides of trending media.

And once the product experience goes live, large scale A-B testing canbe done with active sampling. Safeway stores in Peoria, Ill. and Albany,N.Y. can trial product campaign A for a given package, and stores inGreensboro, N.C. and Santa Barbara Calif. can trial an alternatecampaign B. A week later, the augmentations can be reprogrammed to swapcampaigns. The results from these normative cities can then be compared.If campaign A yields 8% more sales, then program augmentationsnationwide can be programmed that way.

As noted, when a product goes live in stores, retailers may define theirown experiences around the packaging—sometimes using the same functionalmappings as the brand owner, and sometimes with their own customaugmentations.

Consider the Nutrition Facts panel on the cereal box. When a shopperusing the Safeway app selects the Nutrition Facts feature on the box,the annotation may trigger a software module that is defined (orcustomized) by Safeway to compare the Wheaties nutritioninformation—line by line—with other cereals on the shelf. The softwareUI may present the other cereals in an order matching their physicalplacement on the Safeway shelf. If the Wheaties is shelved next toPost's Fruity Pebbles, which is shelved next to Kellogg's FrostedFlakes, then the user may make a sideways swiping gesture to compare theWheaties data first with the Fruity Pebbles data, and then with afurther swipe, with the Frosted Flakes data. The Wheaties data remainsstatic, allowing the user to successively compare its nutritionalinformation with each successive product along the shelf. Swiping in theother direction explores the shelf in the other direction. The user canswipe up or down to move to the shelf above or below, and continue insimilar fashion with other cereals.

A shopper in the Kroger store down the street may use the Kroger app andfind it augments the Nutrition Facts panel of the Wheaties boxsimilarly, but with a comparison UI that is customized to Krogers'particular shelf layout.

An example embodiment for a smartphone, according to the just-describedarrangement, is shown in FIG. 7. The left-most column of data, detailingWheaties nutrition information, is fixed (and was established, e.g., bya user gesture selecting Wheaties cereal as the base product againstwhich others are to be compared). The Fruity Pebbles, Frosted Flakes,and Cheerios (partial) columns to the right, correspond to placement ofthose products to the right of the Wheaties cereal on the store shelf.If, in contrast, the shopper moved the smartphone to the left (e.g., assensed by accelerometers in the device), the Wheaties column ofnutrition information would be presented on the right of the display,with nutrition for neighboring products to the left detailed insuccessive columns to the left.

It will be recognized that the nutrition information detailed in thispresentation is more complete than that provided on the cereal box(e.g., figures are given for omega-3 and omega-6 fatty acids, while suchinformation is not provided in the FIG. 4A information). Suchinformation is available from various web sources, including the website at nutritiondata<dot>self<dot>com. A great deal of further data isavailable to the shopper by scrolling down from the detailed screen ofinformation (e.g., revealing vitamin and mineral information, etc.).

In an alternative embodiment, the Wheaties cereal selected by the useris compared against a cereal imaged by the device camera—without regardto shelf ordering. Such an arrangement is shown in FIG. 8 (in which theright side of the screen serves as a viewfinder for the smartphonecamera, and the left side of the screen compares Wheaties nutrition withthat of the cereal depicted).

Each retailer's augmentations can be stored in a digital assetmanagement system maintained by that retailer. The retailer's appconsults that repository for augmentations first, and then checksrepositories where other augmentations may be found, e.g., a brand's DAMsystem, in accordance with the stored rules—successively examining manyrepositories until augmentation information is found.

If the retailer does not provide its own augmentation for the NutritionFacts feature, the rules for that store's app may invoke theaugmentation defined by the brand. This brand-programmed augmentationmay similarly compare nutrition data for different cereals, but belimited to General Mills cereals (or only Wheaties vs. its competitors'cereals)—not tied to any store shelf layout.

Retailers may offer shoppers enticements to induce them to contributeproduct reviews, and then store such information in a proprietarydatabase—rather than in an open data structure. Thus, reviewscontributed by shoppers using the Safeway app may not be available toshoppers down the street at Kroger's, even if the Safeway app is usedinside Kroger's. (The Safeway app may consult a database that isaccessible only to users connected to the Safeway WiFi network.)

The retailers' augmentations can make use of data about the shopper'spurchase history at that store—allowing finer customization of theaugmentations to the user's particular circumstances. For example, Heinzmay define an augmentation for its 12 ounce catsup bottle that providesa cents-off cross-promotion for its 12 ounce size of companion Heinzmustard. But if the retailer knows the shopper just bought a quart ofmustard the day before yesterday, and knows that the shopper is fond ofgourmet sausages, and knows the shopper buys only whole grain breads,the retailer may instead use an augmentation triggered from the Heinzcatsup bottle to promote its whole grain hotdog buns.

Sometimes a retailer may provide augmentation for CPG hotspots for whichthe brand provides nothing, to introduce gamification aspects into thechore of shopping. Kroger supermarket, for example, may augment selectedinstances of the letter “K” on product packaging with a 50 cent creditat checkout. The rear of the depicted Wheaties box, for example,includes narrative text, “American swimmers have long been known fortheir success in the pool . . . ” If a shopper images the letter “k” in“known,” within a viewing angle of 10 degrees, Kroger supermarkets mayissue a 50 cent checkout credit. The controlling software can limit thenumber of credits a shopper receives per week, e.g., to five, and maypermit a credit from a particular product, e.g., Wheaties, only once inthat period. Only 15% of the “K”s in the store may yield the reward, andthey may be shuffled/re-selected every 5 minutes, to limit gaming of thesystem.

From the foregoing, it will be seen that packaging augmentations aredesirably live and adaptable, not fixed and static.

For such adaptability to be employed for maximum benefit, activemanagement of consumer packaged goods also requires active feedbackabout user interactions

Thus, in accordance with another aspect of the technology, dashboardsoftware is provided for use by brands and retailers, by which they canexplore user interactions with different augmentations under theircontrol. Hourly, daily, and weekly analyses may be available—identifyingwhat bubble selections were made by consumers for given products. A listof stores in which user interactions with particular product(s) areoccurring at that instant may also be presented. For brands, theinformation can be segmented by state, metropolitan media market,retailer, etc. Although the brands may not have information aboutparticular shoppers, the activities of a particular shopper within astore may be discerned, e.g., by a phone hardware identifier that istransmitted as part of the data exchange when an augmentation istriggered. By such information General Mills may learn that, of theOregon customers who interacted with its Cascadian Farms organicproducts, 1.4% also interacted with its Cheerios brand cereals, but only1.1% interacted with its Total brand cereals. In Ohio the numbers arereversed. (It seems the Total brand needs additional promotion amongorganic consumers in Oregon.)

For retailers—with their greater information about customers—the datacan be segmented by age, gender, residence zip code, etc. Importantly,the analyses can extend beyond a particular brand family, to allmerchandise stocked by the retailer. Also, correlations with shoppinghistory data can be performed. Thus, a Safeway store may find that, ofthe women age 18-25 who interact with cartons of fat-free milk, thosewho routinely redeem newspaper coupons (i.e., 28% of such women) end upbuying Post brand cereals in 2.1% of their store visits, and buyingNatures Valley cereals in 2.4% of their visits. In contrast, for thosewho never redeem newspaper coupons (i.e., the other 72%), such women endup buying Post brand cereals in 1.5% of their store visits, and buyingNatures Valley cereals in 1.3% of their visits. Safeway knows that itpublishes equal numbers of coupons for both Post and Natures Valleycereals in the local newspaper, but that its profit margin is 23% higheron Natures Valley products. With some MBA quantitative modeling, Safewaydetermines that it should direct 12% of its newspaper spend on NaturesValley coupons to other forms of media promotion that might be morevisible to the demographic of weight-conscious 18-25 year old females. Aweek later, after taking advantage of a co-op advertising programoffered by Natures Valley, in which the brand pays 90% of up to $5,000in local television ad spending, the dashboard reports that the changehas been successful—the numbers have shifted and profits are up.

A primary metric by which augmentations will be judged is “what ringsthe cash register.” Retailers have the advantage of a tight feedbackloop—they can revise a product's augmentations in the morning, and checkPOS data at the end of the day to see if any statistically significantchange occurred. Brands typically don't have such immediate visibilityinto results, but can still tune augmentations to enhance shopperresponses—just with a longer loop time.

Online retailers, such as Amazon, can identify (e.g., from page-viewdata) products that a shopper looked-at but did not buy. Thisinformation can be used to highlight areas for improvement, e.g.,enhancing the product description, lowering the price, etc. Brick andmortar retailers have lacked this capability. But the present technologycan be used to extend this insight about almost-made-sales into brickand mortar stores. If 60% of consumer interactions with nutrition panelson cereal boxes lead to purchases of the examined products, but Wheatiesis found to lag in this regard with a 30% statistic, then General Mills(or the store) is alerted to a notable deficiency where improvement isneeded. Responsive action—in pricing, product formulation, marketing, orotherwise—can be investigated and pursued.

From the foregoing, it will be evident that shopper interactions withCPG augmentations serve as a probe into shopper sentiment, by whichbrands and retailers can adapt their strategies.

A further aspect of the technology is putting product augmentations outto bid—providing opportunities to communicate with potential consumersto whatever parties value such opportunities the highest.

General Mills isn't likely to offer Kellogg's the chance to define anaugmentation for its front panel Wheaties logo, but it might offer alocal milk vendor, or a regional newspaper, a chance to define anaugmentation elsewhere on the Wheaties box. What's it worth to them?Product profit needn't come only from sales of physical product—higherprofit margins can come from licensing augmentation space.

When a shopper's camera initially recognizes the box of Wheaties, evenbefore the system starts sending responsive information (e.g., 3D modeldata) to the shopper's device, it alerts an ad network about theavailability of a live shopper who will doubtless soon be at acheckout—wallet in hand. Many advertisers would like to make animpression at this propitious moment.

With this notification, the system also makes available relevant contextinformation. This includes demographic profile information about theuser, history information about products with which the user haspreviously interacted (during that store visit, or otherwise),information identifying the previous augmentations that prompted theshopper to action, information about the user's location, etc. Cookiedata on the shopper's portable device can allow access to a dossier ofinformation about the shopper's online activities (and sometimes offlineas well, as detailed in application Ser. No. 14/098,971, filed Dec. 6,2013, now published as 20140164111). The more the ad network knows aboutthe shopper, the more targeted the advertisers' promotions can be, andthe more money the advertisers will pay to make an impression.

Still richer information may be available from the retailer's database,e.g., indicating the shopper's past purchases, frequency of visiting thestore, etc. Depending on the particular implementation, such informationcan be made available to the ad network as well (always subject toappropriate privacy safeguards).

In a blink of an eye, the ad network has evaluated competing offers fromparties wanting to present an advertising impression to the shopper, andhas caused data for one or more winning augmentations to be sent to theshopper device for display. Such augmentation may be mapped to aparticular feature on the product, and probably shown only when thatfeature of the product is depicted on the user device display. Morecommonly, the augmentation is not tied to a particular feature, but isshown for as long as the product is depicted on the shopper's display.

The commercial augmentation may appear as a banner promotion, orscrolling text, across the bottom of a display screen, and may invitethe user to take some action in exchange for a reward. For the Wheatiesexample, one such promotion may invite the user to make a gesture (e.g.,tap a smartphone touch screen) to receive a 25 cent credit off anyhalf-gallon of Alpenrose brand milk purchased while in the store.Another promotion may invite the shopper to watch a promotional 20second video in exchange for a 50 cent credit at checkout. Or thecommercial augmentation may, itself, automatically launch such a videoin a corner of the display screen. The shopper isn't informed in advanceas to the length of the video, but if they tap the video within 2seconds of its conclusion, the shopper is granted a dollar credit atcheckout.

In contrast to the foregoing, sometimes a brand may chose not to sellproduct augmentation opportunities. Instead, it may offer to pay othersto provide a desired augmentation—particularly if it might help sellmore products.

Gatorade, for example, might pay ESPN for the right to present a videoclip of that day's Top Ten Plays as an augmentation on its bottleddrinks. Gatorade markets itself as a drink for athletic competitors, andaffiliating itself with top sports plays may be a prudent marketingcampaign. Whenever a shopper images a Gatorade bottle with a camera, theshopper is presented with a rendering of that day's Top Ten Plays clip.(The user might operate a UI control to expand the rendering frompart-screen to full-screen.) Packages become virtual extensions ofdisplay screens.

If a particular user routinely selects the ESPN augmentation whenviewing a Gatorade bottle, then the system can learn to auto-launch thisaugmentation each time a Gatorade bottle comes into view—withoutrequiring any action by the user.

Sometimes high profile sporting events are embargoed from free networkbroadcast for a period of hours, in accordance with contract terms. SomeOlympic events that took place in London, for example, were embargoedfor broadcast in the US, so that NBC could present them during (US)evening prime time, when the network was able to charge higher rates toadvertisers. During the period before sports coverage is freelyavailable, such content can serve as valuable augmentation for brandswho want to appeal to consumers interested in the embargoed content.(E.g., Coke may invite fans of Olympic diving to interact with a Cokecan, and see highlights from Olympic diving events that just concluded,but that won't be seen on broadcast for several more hours.)

Some of the above augmentations are tailored for in-store presentation(e.g., cents-off coupons and check-out credits). Other augmentations aresuitable at any time (e.g., sporting clips). Thus, a shopper who buys asix-pack of Gatorade may be able to see the day's Top Ten Plays on ESPN,simply by opening the home refrigerator, and viewing a bottle.

(Augmentations that are tailored for in-store presentation can beswapped with other augmentations, for interactions after a productleaves the store. Location information from the user device permits thesystem to determine which augmentation to render.)

Speaking of location, it may be recognized that a product may serve as abeacon that signals location. Store layouts are well-defined—oftenspecified by carefully crafted “planograms” that detail what productsare placed at what locations. If a user's camera senses a box of Tidedetergent, it can be determined that the user is in aisle 6, about 40feet from the north end of the store. Counterpart latitude/longitudecoordinates can also be provided. This location information can be usedin providing known location-based-services and promotions to the user.(A similar outcome can be achieved by using an olfactory sensor ratherthan an image sensor, e.g., signaling that the user is near the colognecounter at Macy's.)

Applicant's published and pending applications, including 20110212717,20110161076, 14/098,971, filed Dec. 6, 2013 (now published as20140164111), and Ser. No. 13/892,079, filed May 10, 2013 (now publishedas 20140164124), provide more information relating to the foregoing,including auctioning on-screen augmentation opportunities, use oflocation and other context in ad presentation, and crowd-sourcingauxiliary content identified by consumers.

In another aspect, the present technology provides virtual communitiesorganized around objects, such as CPGs. Just as each package has adatabase record where associated information is compiled, the packagemay have a web address—around which social interactions can occur andwhere data related to the product can be stored. The internet name spacemay provide a dedicated top level domain (e.g., .object) for thispurpose. An object may have a short, arbitrary address (e.g.,8Kw97G2.object) or a semantically-expressive address can be used (e.g.,Wheaties_Michael_Phelps.object). Or the latter can be used by humans,and can operate simply to redirect the user to the former.

Persons interested in the Wheaties product can link to this web resourceby interacting with a physical box, either at the supermarket or athome. A portal to this web location can be one of the augmentations thatmay be selected when the box is sensed by a user device camera.Alternatively, users can link to the site by interacting with an imageof the product found elsewhere—such as on the web. For example, aright-click of a mouse, or a triple-tap on a touch-screen tablet, canlink from a web page on which a picture of any object appears, to thatobject's web page. Or the user can navigate to the web location by asearch aid, such as Google.

At the web site the user can find a variety of information relating tothe product. By way of example, this information can includecrowd-sourced product commentary, the textual information (e.g.,Nutrition Facts) printed on the packaging, brand-authored promotions ofthe product (video commercials, etc.), links to other media in which theproduct appears (e.g., a Seinfeld episode in which Jason has Wheatiesfor breakfast), map data identifying brick and mortar retailers wherethe product can be purchased, augmentation data authored by the brand,by retailers, and/or by consumers, etc.

Browser software can present alternative views of this universe ofproduct sites, e.g., arranged by manufacturer: General Mills, which hassubsidiary nodes for baking products, cereals, dough/pastries, fruit,ice cream, etc., each of which has subsidiary nodes for differenttrademarked brands (Wheaties, Cheerios, Total, etc.), each of which mayhave subsidiary nodes (Honey Nut Cheerios, Frosted Cheerios, MultiGrainCheerios, etc.). Or the browser may present the sites arranged byproduct types (e.g., with frozen, fresh, and packaged; then—forpackaged—cookies, cereals, pastas, etc.)

Living Packages (and Recycling)

Actively Managed Packages are Living Packages. Living Packages areIntelligent Packages. Intelligent Packages know how to sell themselves,do so most cost effectively, and most of all, they know how to stay outof landfills. Consumers will choose Living Packages over one-time, deadend packages, while the purveyors of those packages will achieve bettermargins in the process. 7 Billion Consumers will demand no less, and yetagain they will drive the economics swiftly toward Actively ManagedPackages.

Recognizable symbols, both proprietary as well as universal (publicdomain), can be one of the underpinnings of a global shift towards muchmore efficient and successful forms of packaging goods. Gary Anderson'snow-familiar Recycling Arrows(http://en<dot>wikipedia<dot>org/wiki/Recycling symbol) will inspire newforms of Living Package symbologies and icons. Anderson's work wasprompted by a competition sponsored by Container Corporation of America.Companies today are even more interested in reducing their ecologicalfootprints, and are still more interested in being seen as leaders inthe green revolution. This, coupled with contemporary consumers' greenemphases and their connected consumption practices, sets the stage forfundamental changes in how 7 Billion people treat our mutual home andits human-material flows. The present technology can play a role.

Actively Managed Packages—ones that magically come alive and offerassistance while the ever-more-wired world simply views them—have theopportunity and the responsibility to advance the culture (and lower thecost) of package-based consumption. FIG. 9 present some thematic notionsof what type of graphic treatments might be possible for consolidatedsymbology, for iconic representations of yet another consumer shifttoward better habits—indicating a visual portal into the world ofproduct recycling.

As with the 1970-era recycle triangles, a whole host of opportunitiesmasquerading as challenges show up in this largely graphic exercise. Theoriginal triangles immediately had to cope with the continuum andnuances of what it actually meant to “recycle.” Is this chicken manurejust going back out into the field, or is there some notional monolithicgovernment program ingesting this object, and lickety-split that samematerial—after some cleaning we hope—is back packaging up the nextversion of the consumable Good? This last sentence is deliberatelyobscure, aiming to underscore the immediate challenges that wererepresented by a bewildering array of paper types; paper versusnon-paper, e.g. aluminum cans vs. non-aluminum . . . on and on. Thedebate was on not just about the details of how the arrows themselvesshould be rendered (and their obscure Mobius-ness), but specifically howvarious treatments of “the symbol” could begin to assist in theunderstanding of the type of recycling intended . . . we could even callit the pre-curser of trying to communicate “which dang bin do I put thisthing in?”

With Actively Managed Packaging, the options for exploiting thesesymbology opportunities for consumer education immediately mushroom,compared to the early 70's opportunities, which by and large werelimited to graphic treatments on packages and on advertisements dealingwith the recycling movement. The main stem of that mushroom is the verycore of “active management”: cloud content, including live humanassistance, can attach and evolve with that package. The graphicallypresented symbology on the Living Package, whatever its form may take,is the gateway to rich content of the “recycling and re-use” variety.Don't know which bin to throw this thing in? . . . Ask the thing itself.

Back to hard core capitalism, shareholders, return on investment and thelike, what's the deal here? The deal couldn't be simpler, summarized bytwo cerebral points, and punctuated by a third common sense point.

Cerebral point 1 is that given an otherwise equal choice, a vastmajority of consumers will choose a responsible option over a dead-endoption. Maybe not everyone, but sales numbers will bear this out—notpredictions in a patent application. Leading to cerebral point 2: theraw material and manufacturing costs for packages, which are designedand engineered within a more diverse supply space of materialsthemselves, will eventually be lower than “extract and discard”approaches. Said another way, this generation of packaging designers andengineers, given the appropriate and the creatively solvable challengeof treating raw material flows as a cost-driven engine, will continue toevolve better material recycling ecosystems, where the hard expensenumbers will be the proof in the pudding. Clearly, some cigar-munchingMr. Potter Capitalist can call BS on this second cerebral point, butgood thing such folks are a dying breed. This is the segue sentence tothe common sense point: it's just the right thing to do and we all knowit.

Clearing the Mr. Potter hurdle then, the description can get back to theinventive details of highly deliberate interactive symbology on consumerpackages . . . that symbology which people will know and expect to “comealive.”

Review

A small sampling of the many inventive arrangements detailed in thisspecification are reviewed below:

One method involves rendering a visible model of a consumer packagedgood from 3D model data stored in a user's portable device. User inputis received, designating a sub-portion of the rendered model as being ofuser interest (e.g., the user is inspecting that portion of the model).A data structure remote from the portable device is consulted toidentify an annotation associated with the designated sub-portion. Thisannotation is then presented to the user. (Fingerprint or watermarkinformation can be derived from the designated sub-portion of therendered model, to identify a location of that sub-portion, for which acorresponding annotation can be identified.)

Still another method involves displaying imagery of a 3D consumerpackaged good, from a first perspective, on a display of a user'sportable device. User input is received, designating a first sub-portionof the displayed good as being of user interest. A data structure isconsulted to identify a first annotation associated with the designatedfirst sub-portion, and this annotation is then presented to the user.(User input may also be received designating a second sub-portion of thedisplayed good as being of interest, and a corresponding secondannotation—identified by consulting a data structure—can be presented tothe user. One of these annotations may be established by a company thatproduced the consumer packaged good, and the other of these annotationsmay not have been established by that company—but rather by anindividual, or by a retailer where the consumer packaged good is offeredfor sale.)

Yet another method includes receiving data indicating selection of afirst sub-portion of a consumer packaged good by a user in a firststore. This user is provided a first annotation that corresponds to thisfirst sub-portion. Data is similarly received indicating selection ofthat same sub-portion of the consumer packaged good by a user in asecond store. This user is provided a second annotation that correspondsto this first sub-portion—which is different than the first annotation.(The first annotation may have been established by an operator of thefirst store, and the second annotation may have been established by anoperator of the second store.)

Another aspect of the technology involves receiving data indicatingselection of a first sub-portion of a consumer packaged good by a firstuser in a first store. A first annotation that corresponds to this firstsub-portion is provided to this first user. Data is also receivedindicating selection of this same first sub-portion of the consumerpackaged good—this time by a second user in the first store. A secondannotation corresponding to this first sub-portion is provided to thissecond user. These first and second annotations are different. (Forexample, the annotations may comprise nutritional information relatingto the good, and the first and second annotations differ due todiffering ages, weights, and/or genders of the first and second users.Or the annotations may differ because the second user has taken actionleading to presentation of nutritional information from a sourcedifferent than the United States government. Or the annotations maydiffer by reason of social network information associated with thesecond user.)

Another aspect of the technology generally concerns identifying pluraldifferent goods on a store shelf, and presenting a visual augmentationfor each of these plural goods to a shopper.

These augmentations may indicate relative sales rankings for differentof the goods, thereby indicating a best seller among the plural goods.Or the augmentations may indicate relative sales rankings for differentof the goods among one or more of a shopper's social network friends,thereby indicating a best seller among these goods from among suchfriends. Or the augmentations may identify social network friends of theshopper who have purchased said goods. Etc.

A further method includes identifying a consumer packaged good on astore shelf, and presenting one or more visual augmentations for thegood to a shopper, where one of these augmentations enables the shopperto communicate with a customer service representative for a company thatproduced or distributed said good.

Another method is similar, but the augmentation enables the user todetermine recycling information related to said good.

Another method is also similar, but the augmentation serves topre-fill—with an identification of the good—a posting to a socialnetwork account of the user, so that the user can more rapidly completesaid posting. (E.g., the augmentation may begin to author a tweet,pre-filled with a hashtag corresponding to the good.)

Still another method includes capturing imagery depicting a consumerpackaged good, using a first user's portable device. Input is receivedfrom the first user, designating a first sub-portion of the consumerpackaged good depicted in the imagery. Annotation information is thenreceived from the first user, concerning this designated sub-portion ofthe good—an annotation that this user wishes to make available to one ormore other users. Information about this first sub-portion, and thisannotation, are stored in a data structure. By such arrangement, theannotation is available from the data structure for presentation to asecond user who designates a second sub-portion of the consumer packagedgood, where the second sub-portion overlaps with the first sub-portion.

A further method includes identifying a first consumer packaged good(e.g., a box of Wheaties cereal), and presenting plural data elementsabout this first good (e.g., different nutrition data) to a shopper on aportable device display. A second consumer packaged good (e.g., a box ofFruity Pebbles cereal) is similarly identified, and plural data elementsabout this second good (corresponding nutrition data) are presented tothe shopper on the display. The information is presented to facilitatecomparison between comparable data elements for the first and secondgoods (e.g., as depicted in FIGS. 7 and 8).

Yet another method involves capturing multi-viewpoint imagery from a 3Dpackage for a consumer good. Positions of plural digital watermark tilesin the captured imagery are identified (e.g., by analyzing the capturedimagery using a hardware processor configured to perform such act).Package map information is generated that defines spatial correspondencebetween artwork on the package and the plural watermark tiles. Thispackage map information is then provided to another party, enabling thatparty to author interactive experiences based on different views of thepackage.

Another aspect of the technology concerns capturing imagery depictingpackage artwork from a 3D package of consumer goods, by resting thepackage on a turntable and rotating the turntable so that imagery fromplural different viewpoints is captured. This captured imagery isprocessed (e.g., by a hardware processor) to discern locations andpayloads of tiled steganographic watermarks encoded in the packageartwork. Resolution information—including at least some of the capturedimagery, and data about the discerned locations and payloads—is storedin a repository. By such arrangement, a package-based interactiveexperience can be authored by reference to a perceptible feature of theartwork, and then re-mapped, using the stored information, to refer toan imperceptible, watermark tile-based feature of the artwork.

A further method involves capturing imagery of an object using acamera-equipped portable device. By reference to information derivedfrom the captured imagery, 3D model information for the object isobtained from a data store. Different views based on the 3D modelinformation are then rendered to a user, responsive to navigation inputreceived from the user. These rendered views are augmented by presentingplural annotations. In particular, a first annotation—but not a secondannotation—is presented when the user navigates to a first view, and asecond annotation—but not the first annotation—is presented when theuser navigates to a second view. (The first annotation can depend onuser context—such as location.)

Another aspect of the present technology concerns a non-transitorycomputer readable medium containing a data structure that plural layersof information associated with a consumer packaged good. These layers ofinformation include at least three from the list: a layer of informationdetailing a visible ink pattern; a layer of information detailing asteganographic digital watermark pattern; a layer of informationdetailing scale-invariant feature points; a layer of informationdetailing a glue pattern; a layer of information detailing 3D modelinformation; a layer of information detailing vendor information; alayer of information detailing a vendor's ink identifier; and a layer ofinformation detailing an augmentation to be presented with a view of theconsumer packaged good. Information on these layers may have beencontributed by different parties, at different times. Some of theinformation may have been transferred from origination materialsauthored by a software tool, and other of the information may have beencaptured from a physical sample of the consumer packaged good.

Another method involves publicizing a game, in which participants arerewarded for interacting with retail products in a store. Aprize-winning interaction is then determined (or confirmed) by referenceto imagery captured by a participant's portable device. For example, thegame may involve imaging certain letters or words found on productpackaging.

A further method includes receiving information about plural URLs, eachdedicated to a respective retail product. Responsive to first userinput, a first view based on the received information is presented, inwhich the information is primarily classified by product producer,rather than product type. Responsive to other user input, a second viewbased on this received information is presented, in which theinformation is primarily classified by product type rather than productproducer.

One method involves an on-screen display of differently-oriented viewsof a product (e.g., a consumer packaged good). When the product isdisplayed at a first orientation, the on-screen display is augmentedwith an annotation that is associated with a first feature on theproduct, and is positioned to the right of the displayed product. Whenthe product is displayed at a second orientation, the display is againaugmented with an annotation associated with the first feature, but nowpositioned to the left of the displayed product. When the product isdisplayed at a third orientation, the product is displayed without theannotation—even though the first feature of the product is visible inthe on-screen display of the product when at the third orientation.

Another method also involves on-screen display of differently-orientedviews of an object. This object includes first and second features on acommon planar face. The method includes augmenting the on-screen displaywith a first annotation associated with the first feature, and with asecond annotation associated with the second feature. When the object isdisplayed at a first orientation, both the first and second annotationsare presented to the right of the displayed object. When the object isdisplayed at a second orientation, both the first and second annotationsare presented to the left of the displayed object. But when the objectis displayed at an orientation between the first and secondorientations, the first annotation is presented to the right of thedisplayed object, and the second annotation is presented to the left ofthe displayed object.

Yet another method also involves an on-screen display ofdifferently-oriented views of an object. Information indicating anorientation at which the displayed object is viewed is determined. Thisinformation is compared with stored reference information, using ahardware processor configured to perform such act. Based on a result ofthe comparison, the on-screen display is augmented by presenting anannotation. Thus, the presentation of the annotation depends both on thedetermined orientation information, and also on stored referenceinformation.

Still another method involves displaying a product with a face at afirst orientation, and augmenting the display with an annotation that isassociated with a first feature on the product. When the product isdisplayed with the face at a second orientation—with the first featureagain visible, displaying the product without the annotation. Theannotation is not displayed in the latter case because an angle betweena surface normal from said face and a viewing direction is outside of apermitted range.

A further method concerns an on-screen display of a product havingplural features, and includes augmenting the display by presenting anannotation associated with a first of said features, and controlling thepresentation based on spatial position of a second of the features,where the first and second features are located at different positions(e.g., different planar surface) on the product.

Still another aspect of the technology concerns imaging a consumerpackaged good (e.g., a drink bottle) with a camera in a user's portabledevice, and in response, presenting a clip of video sports to the useron a display of said portable device. (This clip may be one that is not,and has not been, available on broadcast media, at the time of itspresentation to the user.)

Concluding Remarks Having described and illustrated the principles ofour inventive work with reference to illustrative examples, it will berecognized that the technology is not so limited.

For example, while the detailed embodiments contemplated thataugmentations would be presented on a user device display, this need notbe the case. For example, when a user taps the Nutrition bubble 14 apresented on a mobile phone display in FIG. 1B, responsive graphicinformation may be pushed to a larger format display screen near theuser (e.g., as determined by known location-based-service methods).

While FIG. 6 suggests a simple database record, more elaborate datastructures will likely be used in actual practice. This may includefederated data structures—employing information stored at differentlocations by different parties. Some of the information may be stored ata URL dedicated to the product (e.g., at a top-level .object address).Other information may be stored in a Creative Commons database. Stillother information may be stored in a proprietary database over whichGeneral Mills has control (unlike the previous two).

In the future, it is expected that consumer packaged goods will beserialized. That is, it will be possible to digitally distinguish onebox of Wheaties from a seemingly-identical box. (Patent application61/913,012, filed Dec. 6, 2013, details one such arrangement—employingplural watermarks in tiled arrangement—with the spatial relationshipbetween the tiles varying from box to box as a consequence of theprinting process. RFID chips—sometimes termed NFC chips—could allow suchfunctionality to be achieved now, albeit at a significantly greatercost.) Such item serialization will allow retailers to track particularinstances of goods as they are purchased at check-out terminals. In sucharrangements, when an item is purchased, an entry can be made in adatabase. This entry can cause an augmentation for the item to change,or an augmentation to be enabled. For example, the viewing of ESPNsports programming from Gatorade bottles may be enabled only after ashopper has purchased the bottle. A different payoff may be provided ifthe shopper interacts with the bottle while it is still on the storeshelf.

While reference was made to app software on a smartphone that performscertain of the detailed functionality, in other embodiments thesefunctions can naturally be performed otherwise—including by operatingsystem software on a smartphone, by a server at a social networkingservice, by another smartphone or computer device, distributed betweensuch devices, etc.

While reference has been made to smartphones, it will be recognized thatthis technology finds utility with all manner of devices—both portableand fixed. Tablets, laptop computers, digital cameras, wrist- andhead-mounted systems and other wearable devices, servers, etc., can allmake use of the principles detailed herein. (The term “smartphone”should be construed herein to encompass all such devices, even thosethat are not telephones.)

Particularly contemplated smartphones include the Apple iPhone 5;smartphones following Google's Android specification (e.g., the GalaxyS4 phone, manufactured by Samsung, and the Google Moto X phone, made byMotorola), and Windows 8 mobile phones (e.g., the Nokia Lumia 1020,which features a 41 megapixel camera).

Details of the Apple iPhone, including its touch interface, are providedin Apple's published patent application 20080174570.

The design of smartphones and other computers referenced in thisdisclosure is familiar to the artisan. In general terms, each includesone or more processors, one or more memories (e.g. RAM), storage (e.g.,a disk or flash memory), a user interface (which may include, e.g., akeypad, a TFT LCD or OLED display screen, touch or other gesturesensors, a camera or other optical sensor, a compass sensor, a 3Dmagnetometer, a 3-axis accelerometer, a 3-axis gyroscope, one or moremicrophones, etc., together with software instructions for providing agraphical user interface), interconnections between these elements(e.g., buses), and an interface for communicating with other devices(which may be wireless, such as GSM, 3G, 4G, CDMA, WiFi, WiMax, Zigbeeor Bluetooth, and/or wired, such as through an Ethernet local areanetwork, etc.).

The processes and system components detailed in this specification canbe implemented as instructions for computing devices, including generalpurpose processor instructions for a variety of programmable processors,such as microprocessors (e.g., the Intel Atom, the ARM A5, the QualcommSnapdragon, and the nVidia Tegra 4; the latter includes a CPU, a GPU,and nVidia's Chimera computational photography architecture), graphicsprocessing units (GPUs, such as the nVidia Tegra APX 2600, and theAdreno 330—part of the Qualcomm Snapdragon processor), and digitalsignal processors (e.g., the Texas Instruments TMS320 and OMAP seriesdevices), etc. These instructions can be implemented as software,firmware, etc. These instructions can also be implemented in variousforms of processor circuitry, including programmable logic devices,field programmable gate arrays (e.g., the Xilinx Virtex series devices),field programmable object arrays, and application specificcircuits—including digital, analog and mixed analog/digital circuitry.Execution of the instructions can be distributed among processors and/ormade parallel across processors within a device or across a network ofdevices. Processing of data can also be distributed among differentprocessor and memory devices. As noted, cloud computing resources can beused as well. References to “processors,” “modules” or “components”should be understood to refer to functionality, rather than requiring aparticular form of implementation.

Software instructions for implementing the detailed functionality can beauthored by artisans without undue experimentation from the descriptionsprovided herein, e.g., written in C, C++, Visual Basic, Java, Python,Tcl, Perl, Scheme, Ruby, etc., in conjunction with associated data.Smartphones and other devices according to certain implementations ofthe present technology can include software modules for performing thedifferent functions and acts.

Known browser software, communications software, imaging software, andmedia processing software can be adapted for use in implementing thepresent technology.

Software and hardware configuration data/instructions are commonlystored as instructions in one or more data structures conveyed bytangible media, such as magnetic or optical discs, memory cards, ROM,etc., which may be accessed across a network. Some embodiments may beimplemented as embedded systems—special purpose computer systems inwhich operating system software and application software areindistinguishable to the user (e.g., as is commonly the case in basiccell phones). The functionality detailed in this specification can beimplemented in operating system software, application software and/or asembedded system software.

Different of the functionality can be implemented on different devices.For example, in a system in which a smartphone communicates with acomputer at a remote location, different tasks can be performedexclusively by one device or the other, or execution can be distributedbetween the devices. Extraction of fingerprint and watermark informationfrom imagery is one example of a process that can be distributed in suchfashion. Thus, it should be understood that description of an operationas being performed by a particular device (e.g., a smartphone) is notlimiting but exemplary; performance of the operation by another device(e.g., a remote server), or shared between devices, is also expresslycontemplated.

In like fashion, description of data being stored on a particular deviceis also exemplary; data can be stored anywhere: local device, remotedevice, in the cloud, distributed, etc.

As indicated, the present technology can be used in connection withwearable computing systems, including headworn devices. Such devicestypically include one or more sensors (e.g., microphone(s), camera(s),accelerometers(s), etc.), and display technology by which computerinformation can be viewed by the user—either overlaid on the scene infront of the user (sometimes termed augmented reality), or blocking thatscene (sometimes termed virtual reality), or simply in the user'speripheral vision. A headworn device may further include sensors fordetecting electrical or magnetic activity from or near the face andscalp, such as EEG and EMG, and myoelectric signals—sometimes termedBrain Computer Interfaces, or BCIs. (A simple example of a BCI is theMindwave Mobile product by NeuroSky, Inc.) Exemplary wearable technologyis detailed in patent documents U.S. Pat. No. 7,397,607, 20100045869,20090322671, 20090244097 and 20050195128. Commercial offerings, inaddition to the Google Glass product, include the Vuzix Smart GlassesM100, Wrap 1200AR, and Star 1200XL systems. An upcoming alternative isaugmented reality contact lenses. Such technology is detailed, e.g., inpatent document 20090189830 and in Parviz, Augmented Reality in aContact Lens, IEEE Spectrum, September, 2009. Some or all such devicesmay communicate, e.g., wirelessly, with other computing devices (carriedby the user or otherwise), or they can include self-contained processingcapability. Likewise, they may incorporate other features known fromexisting smart phones and patent documents, including electroniccompass, accelerometers, gyroscopes, camera(s), projector(s), GPS, etc.

Embodiments of the present technology can also employ neuromorphicprocessing techniques (sometimes termed “machine learning,” “deeplearning,” or “neural network technology”). As is familiar to artisans,such processors employ large arrays of neuron-likeelements—interconnected to mimic biological synapses. Such processorsemploy programming that is different than the traditional, von Neumann,model. In particular, connections between the circuit elements areweighted according to correlations in data that the processor haspreviously learned (or been taught). When a pattern of data (e.g., a setof audio, image or other sensor data) is applied to the processor (i.e.,to inputs of several of the circuit elements), certain nodes may spikewhile others remain relatively idle. Each of these nodes may serve as aninput to plural other circuit elements, triggering further spiking incertain other nodes—a chain reaction that ultimately provides signals tooutput nodes to indicate the results of the neuromorphic processing. (Inaddition to providing output signals responsive to the input data, thisprocess can also serve to alter the weightings, training the network tobetter respond to certain patterns that it has seen (i.e., processed)before.) Such techniques are well suited for pattern recognitionapplications, among many others.

Additional information on such techniques is detailed in the Wikipediaarticles on “Machine Learning,” “Deep Learning,” and “Neural NetworkTechnology,” as well as in Le et al, Building High-Level Features UsingLarge Scale Unsupervised Learning, arXiv preprint arXiv:1112.6209(2011), and Coates et al, Deep Learning with COTS HPC Systems,Proceedings of the 30th International Conference on Machine Learning(ICML-13), 2013. These journal papers, and then-current versions of the“Machine Learning” and “Neural Network Technology” articles, areattached as appendices to copending patent application 61/861,931, filedAug. 2, 2013.

Various references were made to context. The artisan will understandthat context refers to any information useful in characterizing thesituation of an entity (an entity being a person, place or object thatis considered relevant to an interaction between a user and anapplication, including the user and application themselves).

Context information can be of many sorts, including computing context(network connectivity, resource availability, processor type, CPUcontention, etc.), user context (user profile, location, actions,preferences, nearby friends, social network(s) and situation, etc.),physical context (e.g., lighting, noise level, traffic, sensed sounds,recognized speech, etc.), temporal context (time of day, day, month,season, etc.), history of the above, etc.

Another taxonomy of context progresses from simple and concrete, tocomplex and abstract, starting with location, then physical context (asdetermined by sensors, e.g., device orientation and motion, temperature,infrared, video, 3D ambient audio, ultrasonic, humidity, gases and otherchemical), then user or device actions (e.g., writing, talking, reading,searching, navigating, pointing), then proximities (e.g., to people,vehicles, buildings, perimeters, jurisdictions, other devices), thensomatic (e.g., live datastreams of biometric information), then datafeeds (e.g., subscriptions and RSS feeds, social network follows, alertsand updates), then emergent live data (e.g., from external sources, suchas calls, text, email, weather forecasts), and finally n-dimensionalcontext history—encompassing some or all of the foregoing.

Context arrangements suitable for use with the present technology arefurther detailed in the documents incorporated by reference.

While many of the illustrative embodiments made reference to digitalwatermarking for content identification, in most instancesfingerprint-based content identification (i.e., recognition of inherent,rather than added, patterns) can be used instead.

The techniques of digital watermarking are presumed to be familiar tothe artisan. Examples are detailed, e.g., in Digimarc's patent documentsU.S. Pat. Nos. 6,614,914, 6,590,996, 6,122,403, 20100150434 and20110274310, as well as in pending application Ser. No. 13/946,968,filed Jul. 19, 2013 (now published as 20140052555), and 61/909,989,filed Nov. 27, 2013. Such watermarks are commonly imperceptible, meaningthey are not noticeable to a viewer examining watermarked packaging froma typical viewing distance (e.g., 20 inches). Spot colors, as aresometimes found on packaging, can be watermarked by leaving tiny voidsin the printing to subtly change the luminance or chrominance. Othertechniques for watermarking of spot colors are detailed in U.S. Pat. No.6,763,124 and application Ser. No. 13/975,919, filed Aug. 26, 2013 (nowU.S. Pat. No. 9,449,357).

Fingerprint-based content identification techniques are also well known.SIFT, SURF, ORB and CONGAS are some of the most popular algorithms.(SIFT, SURF and ORB are each implemented in the popular OpenCV softwarelibrary, e.g., version 2.3.1. CONGAS is used by Google Goggles for thatproduct's image recognition service, and is detailed, e.g., in Neven etal, “Image Recognition with an Adiabatic Quantum Computer I. Mapping toQuadratic Unconstrained Binary Optimization,” Arxiv preprintarXiv:0804.4457, 2008.) Still other fingerprinting techniques aredetailed in patent publications 20090282025, 20060104598, WO2012004626and WO2012156774 (all by LTU Technologies of France).

Yet other fingerprinting techniques are variously known as Bag ofFeatures, or Bag of Words, methods. Such methods extract local featuresfrom patches of an image (e.g., SIFT points), and automatically clusterthe features into N groups (e.g., 168 groups)—each corresponding to aprototypical local feature. A vector of occurrence counts of each of thegroups (i.e., a histogram) is then determined, and serves as a referencesignature for the image. To determine if a query image matches thereference image, local features are again extracted from patches of theimage, and assigned to one of the earlier-defined N-groups (e.g., basedon a distance measure from the corresponding prototypical localfeatures). A vector occurrence count is again made, and checked forcorrelation with the reference signature. Further information isdetailed, e.g., in Nowak, et al, Sampling strategies for bag-of-featuresimage classification, Computer Vision—ECCV 2006, Springer BerlinHeidelberg, pp. 490-503; and Fei-Fei et al, A Bayesian HierarchicalModel for Learning Natural Scene Categories, IEEE Conference on ComputerVision and Pattern Recognition, 2005; and references cited in suchpapers.

Optical character recognition (OCR) can be similarly employed toidentify objects, and can also serve to provide anchor points to whichannotations may be spatially referenced.

Use of such identification technologies to obtain object-relatedmetadata is familiar to artisans and is detailed, e.g., in theassignee's patent publication 20070156726, as well as in publications20120008821 (Videosurf), 20110289532 (Vobile), 20110264700 (Microsoft),20110125735 (Google), 20100211794 and 20090285492 (both Yahoo!).

Linking from watermarks (or other identifiers) to corresponding onlinepayoffs is detailed, e.g., in Digimarc's patents U.S. Pat. Nos.6,947,571 and 7,206,820.

Applicant's other work that is relevant to the present technologyincludes that detailed in pending patent application Ser. No.13/651,182, filed Oct. 12, 2012 (now published as 20140106710), Ser. No.13/789,126, filed Mar. 7, 2013 (now published as 20140119593), Ser. No.13/892,079, filed May 10, 2013 (now published as 20140164124),61/838,165, filed Jun. 21, 2013, and Ser. No. 14/074,072, filed Nov. 7,2013 (now published as 20140258110), and published applications20100228632, 20120218444, 20120046071, 20120300974, 20120224743,20120214515, 20130097630, 20130311329, 20130314541.

Some of the third party work that is relevant to the present technologyincludes that detailed in published applications 20130250048,20130290106, 20130246182, 20120215640, 20120233003, 20130127980 and20070106721.

This specification has discussed several different embodiments. Itshould be understood that the methods, elements and concepts detailed inconnection with one embodiment can be combined with the methods,elements and concepts detailed in connection with other embodiments.While some such arrangements have been particularly described, many havenot—due to the large number of permutations and combinations. Applicantsimilarly recognizes and intends that the methods, elements and conceptsof this specification can be combined, substituted and interchanged—notjust among and between themselves, but also with those known from thecited prior art. Moreover, it will be recognized that the detailedtechnology can be included with other technologies—current andupcoming—to advantageous effect. Implementation of such combinations isstraightforward to the artisan from the teachings provided in thisdisclosure.

While this disclosure has detailed particular ordering of acts andparticular combinations of elements, it will be recognized that othercontemplated methods may re-order acts (possibly omitting some andadding others), and other contemplated combinations may omit someelements and add others, etc.

Although disclosed as complete systems, sub-combinations of the detailedarrangements are also separately contemplated (e.g., omitting various ofthe features of a complete system).

While certain aspects of the technology have been described by referenceto illustrative methods, it will be recognized that apparatusesconfigured to perform the acts of such methods are also contemplated aspart of applicant's inventive work. Likewise, other aspects have beendescribed by reference to illustrative apparatus, and the methodologyperformed by such apparatus is likewise within the scope of the presenttechnology. Still further, tangible computer readable media containinginstructions for configuring a processor or other programmable system toperform such methods is also expressly contemplated.

The present specification should be read in the context of the citedreferences. Those references disclose technologies and teachings thatthe applicant intends be incorporated into embodiments of the presenttechnology, and into which the technologies and teachings detailedherein be incorporated.

To provide a comprehensive disclosure, while complying with thestatutory requirement of conciseness, applicantincorporates-by-reference each of the documents referenced herein. (Suchmaterials are incorporated in their entireties, even if cited above inconnection with specific of their teachings.) These references disclosetechnologies and teachings that can be incorporated into thearrangements detailed herein, and into which the technologies andteachings detailed herein can be incorporated. The reader is presumed tobe familiar with such prior work.

The claims submitted with this application address just a small fractionof the patentable inventions disclosed herein. Applicant expects manymore, and broader, claims will be issued from this patent family.

In view of the wide variety of embodiments to which the principles andfeatures discussed above can be applied, it should be apparent that thedetailed embodiments are illustrative only, and should not be taken aslimiting the scope of the invention. Rather, applicant claims as theinvention all such modifications as may come within the scope and spiritof the following claims and equivalents thereof.

1-65. (canceled)
 66. A method employing a camera- and display-equippedportable apparatus, to present imagery of a 3D object as captured bysaid apparatus from a viewpoint, together with an augmentationassociated with said object, the 3D object comprising first and secondnon-coplanar faces, the method comprising the acts: from first imageryof the object captured by the apparatus, discerning a viewpoint betweenthe apparatus and the first face of the 3D object; when said discernedviewpoint between the apparatus and the first face of the 3D object hasa first value, not presenting an augmentation associated with the secondface of the 3D object, due to said first value not being within apermitted range; and when said discerned viewpoint between the apparatusand the first face of the 3D object has a second value different thanthe first, presenting said augmentation associated with the second faceof the 3D object, due to said second value being within a permittedrange.
 67. A method employing a camera- and display-equipped portableapparatus, to present imagery of a retail item, together with first andsecond different augmentations associated with the retail item, themethod comprising the acts: from first imagery of the item captured bythe apparatus, discerning an identity of the item; based on thediscerned identity, recalling 3D model data for the item from a memory;presenting a first rendering of the item on the display of theapparatus, based on the 3D model data, the first rendering having afirst viewpoint, and presenting the first augmentation with said firstrendering; in response to user manipulation of a user interface,presenting a second rendering of the item on the display of theapparatus, the second rendering having a second viewpoint different thanthe first rendering; checking a datum characterizing the secondviewpoint against stored data indicating a range of viewpointsassociated with the second augmentation, and based on an outcome of saidchecking, determining whether to present the second augmentation withsaid second rendering.
 68. A method employing a camera- anddisplay-equipped portable apparatus, to present imagery of a retail itemas captured by said apparatus from a viewpoint, together with anaugmentation associated with said retail item, the method comprising theacts: from first imagery of the retail item, discerning an identity ofthe retail item; based on said discerned identity of the retail item,determining a first augmentation associated with said retail item, andpresenting the first augmentation to the user; and in response to userselection of the first augmentation, software of the apparatus authoringa social media communication that includes a hashtag corresponding tosaid retail item.
 69. The method of claim 68 that further includes: fromsaid first imagery, also discerning a viewpoint between the apparatusand the retail item, with which the first imagery was captured by saidapparatus; based on said discerned identity of the retail item,determining data indicating a range of permitted viewpoints for whichthe first augmentation should be displayed with captured imagery of theretail item; and based on said discerned viewpoint, and said range ofpermitted viewpoints, determining that the first augmentation should bepresented to a user.